19“*  ICCRTS 

“C2  Agility:  Lessons  Learned  from  Research  and  Operations” 
The  Situation  Awareness  Weighted  Network  (SAWN)  Model 

Topics:  2,  5 


Irena  Ali,  Alex  Kalloniatis,  Elizabeth  Kohn,  Phuong  La, 
Iain  Macleod,  Timothy  Neville  and  Mathew  Zuparic 

POC:  A.  Kalloniatis 


Defence  Science  &  Technology  Organisation 
Defence  Establishment  Fairbairn 
24  Fairbairn  Avenue 
Canberra,  ACT 
Australia 

Telephone:  +61  2  612  86468 
E-Mail:  Alexander.Kalloniatis @ dsto.defence. gov.au 


0 


Form  Approved 
0MB  No.  0704-0188 


Report  Documentation  Page 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 


1.  REPORT  DATE 

JUN  2014 

2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2014  to  00-00-2014 

4.  TITLE  AND  SUBTITLE 

The  Situation  Awareness  Weighted  Network  (SAWN)  Model 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Defence  Science  &  Technology  Organisation, Defence  Establishment 
Fairhairn,24  Fairhairn  A  venue, Canberra,  ACT  Australia, 

8.  PEREORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

Presented  at  the  18th  International  Command  &  Control  Research  &  Technology  Symposium  (ICCRTS) 
held  16-19  June,  2014  in  Alexandria,  VA.  U.S.  Government  or  Federal  Rights  License 


14.  ABSTRACT 


15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIEICATION  OE: 

17.  LIMITATION  OE 
ABSTRACT 

18.  NUMBER 
OE  PAGES 

19a.  NAME  OE 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

38 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


The  Situation  Awareness  Weighted  Network  (SAWN)  Model 

Irena  Ali,  Alex  Kalloniatis^,  Elizabeth  Kohn,  Phuong  La,  Iain  Macleod,  Timothy  Neville  and 
Mathew  Zuparic  (Joint  and  Operations  Analysis  Division) 

We  introduce  a  novel  methodology  to  examine  how  a  distributed  organisation  of  military  staff 
who  feed  a  Common  Operating  Picture  (COP)  generates  Situation  Awareness  (SA),  a  critical 
component  for  achieving  agility.  The  model  we  propose  combines  two  dominant  scientific 
models  of  SA,  by  Endsley  and  by  Stanton  et  al.  respectively.  The  Endsley  model  posits  that 
SA  can  ascend  up  three  levels:  Perception,  Comprehension  and  Projection.  The  Stanton 
Distributed  model  proposes  that  SA  exists  across  a  social  and  semantic  network  of  people  and 
information  objects  in  activities  connected  across  a  set  of  tasks. 

The  fusion  of  these  we  call  the  Situation  Awareness  Weighted  Network  (SAWN).  The  output 
of  SAWN  is  a  representation  as  a  weighted  bipartite  network  of  the  interaction  between 
people  (‘human  nodes’)  and  information  artefacts  such  as  documents  and  system  displays 
(‘product  nodes’);  link  weights  represent  the  Endsley  levels  of  SA  that  individuals  acquire 
from  or  provide  to  information  objects  and  other  individuals.  We  will  show  how  data  can  be 
collected  through  surveys  or  structured  interviews  against  both  ‘steady-state’  and  crisis 
scenarios  or  exercises;  in  the  latter  case  task  models  may  be  developed  which  provide  the 
context  for  such  data  collection.  A  set  of  twelve  survey/interview  questions  graduate  the 
Endsley  model  into  finer  increments  of  SA.  Eor  each  of  these  a  respondent  is  asked  about  use 
and  consumption  of  products,  and  interaction  (including  face-to-face,  email,  chat,  phone, 
video  conference)  with  another  human  node  for  the  benefit  of  one  or  the  other. 

The  resulting  network  representation  reveals  sources  and  sinks  of  S  A,  the  means  by  which 
fusion  increases  levels  of  SA  and  how  S  A  flows  through  a  C2  organisation.  Quantitatively, 
the  network  representation  naturally  leads  to  new  metrics  for  SA  flow  such  as  nestedness  and 
the  weight  gradient  along  paths  in  the  graph.  We  illustrate  the  method  with  aggregated  data. 
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1.  Introduction 

The  C2  community  understands  well  that  agility  -  the  property  of  being  robust,  resilient, 
flexible  and  adaptable  [NATO  2006]  -  is  critical  in  circumstances  where  organisations  must 
quickly  respond  to  rapidly  developing  situations  with  multiple  elements  whose  connections 
are  opaque.  Situation  Awareness  (SA)  -  of  individuals  and  the  organisations  in  which  they 
perform  their  roles  -  is,  in  turn,  a  key  requirement  for  achieving  that  agility;  there  is  little  time 
for  hierarchical  handling,  processing  and  authorised  dissemination  of  information.  Thus 
agents  in  the  system  must  each  be  able  to  understand  what  is  happening  and  make  judgements 
as  to  what  may  happen  next  and  enable  others  to  do  the  same.  For  this  reason,  a  model  of  SA 
that  takes  into  account  individual  and  distributed  cognition  is  essential.  This  paper  proposes 
such  a  model  by  unifying  a  number  of  existing  approaches  to  SA.  The  model  we  arrive  at  is 
ideally  suited  to  an  organisational  context  of,  for  example,  military  staff  who  maintain  a 
Common  Operating  Picture  (COP)  within  the  J2  (Intelligence)  and  J3  (Operations)  functions 
in  response  to  a  rapidly  unfolding  crisis. 

The  context  of  this  research  is  a  recently  completed  study  with  an  Australian  Defence  Force 
military  headquarters  organisation.  We  present  in  this  paper  an  aggregation  of  the  data  set 
collected  in  the  study  such  that  the  broad  characteristics  of  the  set  are  evident  but  individual 
staff,  roles,  units  and  their  inputs/outputs  are  not  identified.  However,  key  to  the  context  of 
our  study  are  the  following: 

•  Individuals  or  teams  -  entities  we  will  say  -  have  their  attention  drawn  to  some 
element  either  in  their  field  of  view  or  through  some  system  display,  and/or  must 
develop  an  understanding  of  the  current  actions  of  those  elements  and/or  what  may 
happen  into  the  future  -  we  deliberately  here  draw  on  the  language  of  one  of  the  well- 
known  models  of  SA,  that  of  Mica  Endsley  [1985].  We  say  ‘and/or’  because  these 
entities  may  not  have  access  to  sufficient  information  to  enable  them  to  understand 
the  current  actions  or  anticipate  the  future. 

•  These  entities  must  then  communicate  that  information,  either  as  an  information 
artefact  (document,  slide,  update  of  a  system  display)  or  through  personal  interaction 
by  some  medium  (face-to-face,  phone,  email,  video,  chat). 

•  Other  entities,  with  access  to  other  information  or  some  of  the  aforementioned 
artefacts  or  interactions,  may  achieve  the  understanding  or  anticipation  discussed 
above  and  in  turn  communicate  that  value-added  information. 

•  Finally,  some  of  the  aforementioned  entities  or  still  others  may  formally  be 
responsible  for  initiating  actions  based  on  the  anticipated  actions  of  the  elements. 

We  see  that  these  aspects  are  all  present  in  a  diversity  of  ‘C2  Problems’,  from  the  threat  of  a 
military  or  terrorist  adversary  to  the  consequences  of  a  natural  disaster,  and  in  a  diversity  of 
‘C2  Approaches’,  from  hierarchy  to  complete  network  centricity.  It  is  the  coexistence  of  these 
that  have  warranted  our  fusion  of  the  two  dominant  models  of  SA.  One  is  the  model  that  is 
well  suited  to  the  case  of  individual  cognition  and  with  some  extension  to  ‘team  SA’  [Endsley 
1985,  1987,  1988].  The  other  is  known  as  ‘Distributed  SA’  [Walker  et  al.  2006,  Salmon  et  al. 
2010],  which  sees  SA  invested  in  a  system  of  networked  individuals  (who  may  not  belong  to 
a  team  at  all)  and  information  artefacts.  For  reasons  to  become  clear,  we  have  named  our 
approach  the  Situation  Awareness  Weighted  Network  (SAWN)  model.  We  will  define  this 
model  and  provide  a  systematic  data  collection  methodology,  itself  based  on  well-established 
methods  in  the  scientific  literature. 

We  emphasise  that  the  intent  of  this  paper  is  to  explain  the  model  and  methodology  and  show 
how  insights  into  the  effectiveness  of  the  generation  of  SA  by  a  distributed  system  of  people 
and  tools  can  be  gained  in  order  to  derive  a  ‘future’  C2  structure.  The  intent  is  not  to  make, 
here,  statistically  valid  judgements  of  an  existing  system.  Therefore,  in  this  paper  we  will  only 
sufficiently  specify  the  data  such  that  the  representations  we  show  are  reproducible. 
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The  study  began  with  a  comprehensive  familiarisation  with  the  setting  through  analysis  of 
Standing  Operating  Procedures  (SOPs),  observations  of  headquarters  activities  across  all  time 
periods  in  the  24  hour  cycle,  including  attendance  at  key  daily  briefs,  and  unstructured 
interviews  with  staff.  At  the  same  time  we  examined  the  literature  on  SA  to  determine 
whether  existing  models  and  measures  could  be  applied  and  to  provide  data  to  address  the 
study  question.  The  overall  study  approach  is  depicted  in  Figure  1,  to  be  explained  further 
through  the  paper. 


Figure  1  Overall  study  approach 


In  the  next  section  we  expand  on  the  existing  SA  models  drawn  together  into  our  proposal. 

We  then  elaborate  on  SAWN  and  the  data  collection  methodology  at  the  heart  of  Figure  1.  A 
core  part  of  this  methodology  requires  an  articulation  of  requirements  for  SA  by  planners  and 
operators,  which  we  describe.  We  then  show  the  results  of  such  a  collection  for  a  hypothetical 
context  and  discuss  the  broad  applicability  of  the  approach. 

2.  Existing  Models  of  SA 

SA  has  been  subject  to  considerable  research  in  the  military  and  scientific  communities  over 
the  last  few  decades.  There  are  numerous  SA  models  and  measurements  described  in  the 
academic  literature  dealing  with  individual,  team,  shared,  and  distributed  situation  awareness 
[Endsley  1985,  Adams  et  al.  1995,  Salas  et  al.  1995,  Banburry  et  al.  2004,  Houghton  et  al. 
2008,  Salmon  et  al.  2010,  Walker  et  al.  2010,  Hew  201 1].  Most  of  the  models  discussed  in  the 
literature  are  in  agreement  that  SA  is  a  process  by  an  individual  human  combining,  integrating 
and  interpreting  information  from  the  environment,  system  displays  and  other  humans.  The 
two  models  mentioned  in  the  introduction  are  distinguished  by  being,  contrastingly,  an 
individual  operator  based  model,  that  of  Endsley  [1985,  1987,  1988,  2000],  and  a  distributed 
system  model  [Stanton  et  al.  2006,  Salmon  et  al.  2009,  2010,  Walker  et  al.  2010]. 

Endsley  SA 

Endsley  [1985]  defines  SA  as  a  level  of  awareness  that  an  individual  has  of  a  given  situation, 
an  understanding  of  and  knowledge  of  what  is  going  on.  Accordingly,  SA  is  a  product  of 
situation  assessment  comprising  three  levels:  the  perception  of  the  elements  in  the 
environment  within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning,  and  the 
projection  of  their  status  in  the  near  future  [Endsley  1985:  p36].  These  levels  are  depicted  in 
Eigure  2. 
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Action 

■  Decision 

Figure  2  The  Endsley  model  of  SA 


SA  Level  I  is  where  data  is  merely  perceived  with  no  further  processing.  Level  II  involves 
interpretation  of  Level  I  data,  enabling  an  individual  to  understand  its  relevance  in  relation  to 
the  task  performed  and  goals  to  be  attained.  It  is  where  an  operator  forms  a  holistic  picture  of 
the  operational  environment  and  comprehends  the  significance  of  objects  and  events  in  that 
environment  [Endsley  1985].  Based  on  a  combination  of  SA  Levels  I  and  II,  together  with 
experience  and  knowledge,  operators  may  then  forecast  likely  future  states  for  the  situation; 
this  gives  SA  Level  III.  A  key  assumption  in  this  three-level  model  is  the  pivotal  role  of 
mental  models  on  the  development  and  maintenance  of  SA.  Mental  models,  resulting  from 
training  and  experience,  are  used  to  facilitate  the  achievement  of  SA  by  identifying 
elements/events  in  the  environments  (Level  I),  then  integrating  them  to  gain  understanding  of 
their  meaning  (Level  II),  and  finally  for  generating  likely  future  states  (Level  III). 

The  measurement  techniques  associated  with  this  model  have  been  used  and  validated 
extensively,  notably:  the  Situation  Awareness  Global  Assessment  Technique  (SAGAT)  where 
‘freeze -probe’  techniques^  are  applied;  the  Situation  Present  Assessment  Model  (SPAM) 
where  probes  are  administered  in  ‘real-time’^;  the  Situation  Awareness  Rating  Technique 
(SART)  eliciting  subjective  assessments  of  SA  and  usually  administered  post  trial;  and 
observer  rating  where  domain  experts  observe  and  rate  participants  performing  a  task  under 
study. 

The  Endsley  model  has  been  widely  accepted  as  a  result  of  its  positioning  of  SA  between 
cognitive  science  and  individual/personal  experience. 

Distributed  SA 

A  difficulty  with  the  Endsley  individual  operator  model  arises  when  it  is  applied  to 
collaborative  systems.  Moreover,  the  measures  associated  with  this  model,  apart  from  relying 
on  subject  matter  experts  and  being  rather  intrusive,  are  most  suitable  for  assessing  SA  in 
stable  environments  and  where  SA-related  elements  can  be  pre-defined,  namely  where 
‘ground  truth’  is  known  [Salmon  et  al.  2009,  Saner  et  al.  2009].  However,  when  the  tasks  are 


^  A  task  is  randomly  frozen  and  all  displays/screens  are  blanked.  A  set  of  SA  queries  regarding  the  current 
situation  at  the  time  of  the  freeze  is  applied  and  participants  are  required  to  answer  questions  based  on  their 
knowledge  and  understanding  of  the  situation  at  that  point.  Responses  are  compared  to  the  state  of  the  system  at 
the  point  of  freeze  and  the  overall  score  is  calculated  at  the  end  of  trial. 

^  No  freezing  of  task  under  analysis  and  SA  related  queries  are  administered  on-line  while  an  operator  is 
performing  a  task.  The  queries  are  developed  in  advance  by  a  subject  matter  expert. 
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dynamic,  collaborative  and  changeable  and  the  outcomes  are  not  known,  then  a  different  non¬ 
linear  approach  is  more  suitable.  The  premise,  then,  of  the  Distributed  Situation  Awareness 
(DSA)  model  is  that  cognition  is  distributed  and  emergent  in  systems  comprising  interacting 
human  and  technological  agents  [Salmon  et  al.  2010].  DSA  thus  enables  the  assessment  of  SA 
in  complex  environments  where  situations  change  dynamically  and  where  team  members 
possess  unique  and  compatible  portions  of  awareness.  At  the  systemic  level,  awareness  is 
distributed  across  different  human  and  technological  agents  involved  in  collaborative  tasks. 

At  the  team  level,  members  experience  a  situation  differently  depending  on  their  knowledge, 
experience,  training,  task,  role  and  goals.  SA  involves  interactions  between  operators  via  a 
social  network,  between  operators  and  the  technology/artefacts  through  a  propositional  (or 
knowledge)  network,  and  between  tasks  and  the  information  they  require  to  perform  them 
connected  in  a  task  network.  SA  is  associated  with  individual  agents  but  may  not  reside 
within  them  as  it  is  borne  out  of  interactions  between  them  [Salmon  et  al.  2010].  The 
advantage  of  the  DSA  model  is  that  it  accounts  for  different  information  elements  for  a  given 
task  in  terms  of  their  usage,  sharing,  and  ownership.  It  allows  representation  of  the  changing 
nature  of  tasks  and  task  phases  in  response  to  a  dynamic  operational  environment. 

In  more  detail,  propositional  networks  are  created  by  defining  the  concepts  or  information 
elements  and  their  relationships,  expressing  what  needs  to  be  known  for  the  performance  of 
tasks  [Anderson  1983,  Crandall  et  al.  2006,  Houghton  et  al.  2008,  Salmon  et  al.  2010].  The 
information  elements  for  a  given  task  are  discerned  either  by  observation  or  via  the  Critical 
Decision  Method  (CDM)  [Klein  1989,  Hoffman  et  al.  1998].  Propositional  networks  thus 
build  a  team  SA  picture  from  all  the  pieces  that  individuals  hold.  Each  element  in  a  defined 
relationship  should  be  built  from  a  proposition  that  can  be  either  true  or  false  and  for  which  it 
is  critical  that  there  is  a  name  for  the  relationship. 

Why  unify  the  models? 

Our  initial  observations  in  the  study  clearly  recommended  both  models  of  SA.  The  distributed 
property  of  the  headquarters  staff,  both  as  a  team  and  with  their  interaction  in  a  wider 
community  and  with  the  products  they  generate,  was  evident.  So  too  was  the  effort  of 
individual  subjects  in  analysis  of  sensor  data  to  understand  and  anticipate  actions  of  platforms 
and  people  to  build  the  COP;  adding-value  was  something  our  operators  consciously  sought 
to  do.  That  the  Endsley  and  DSA  models  may  be  unified  was  first  proposed  by  our  colleague, 
Patrick  Hew.  While  addressing  the  integration  of  human  and  machine  elements  in  SA  [Hew 
201 1],  he  proposed  a  model  built  on  Time  Coloured  Petri  Nets  (capturing  the  distributed  SA 
aspect)  consisting  of  a  hierarchy  of  agents  identified  by  the  terms  Data,  Track,  Actions 
(capturing  the  three  Endsley  SA  levels).  Such  an  output  is  built  based  on  a  scientist’s 
understanding  of  the  application  domain  (for  example,  sensor-shooter  systems  and 
interactions  in  coordinated  Eires)  -  much  like  Business  Process  Modelling  (BPM).  In  this 
respect  the  approach  diverges  from  the  spirit  of  Social  Network  Analysis  in  which  the 
network  is  directly  built  on  raw  interaction  data.  It  therefore  suffers  from  some  of  the 
subjectivity  of  BPM,  where  the  scientist  must  interpret  how  the  business  is  performed  and 
generate  a  stylised  model.  Eor  that  same  reason,  developing  models  out  of  data  is  also 
manually  intensive.  This  motivates  our  approach  in  SAWN  to  have  ‘social’  and  ‘information 
artefact  ’  interactions  captured  in  the  same  data  collection  instrument. 

3.  SAWN 

Traditional  Social  Network  Analysis  captures  the  interaction  patterns  of  individuals  in  some 
form  of  exchange,  which  may  be  purely  social  or  business  in  nature.  Individuals  are  nodes  in 
a  network  and  the  interaction  is  a  link  between  nodes.  SAWN  expands  on  this  by  including 
nodes  representing  information  artefacts:  documents,  PowerPoint  slides,  signals  and  whole  or 
parts  of  system  displays  used.  Individuals  thus  transact  information  between  themselves 
and/or  information  artefacts.  These  transactions  are  represented  as  links.  This  much  combines 
the  propositional,  social  and  task  network  aspects  of  DSA.  In  distinction  to  DSA,  SAWN  sees 
weights  applied  to  the  links  -  as  line  thickness  or,  in  our  case,  colours  -  where  the  weights 
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indicate  the  SA  level,  according  to  the  Endsley  model.  Finally,  we  can  distinguish  separate 
networks  for  the  pulling  (or  consumption)  and  pushing  (production)  of  information.  A  simple 
example  of  SAWN  is  shown  in  Figure  3  where  individuals  A,  B,  C,  D  interact  with  each  other 
and  with  information  artefacts  X,  Y,  Z,  W.  But  through  this  transaction  A  has  delivered  low 
SA  (thin  line)  to  B,  but  B  correlating  this  with  information  product  Z  has  delivered  higher  SA 
to  C  who  finally,  through  access  to  artefact  W,  provides  the  highest  SA  level  to  D. 


Figure  3  Representation  of  a  Situation  Awareness  Weighted  Network 
4.  C2  context 

The  military  headquarters  in  our  study  works  broadly  in  the  lower  operational  to  upper 
tactical  spectrum  of  command,  structured  using  the  Common  J  Staff  System  (CJSS), 
interacting  with  other  Joint  Task  Forces  (JTFs),  coalitions  (US,  UK,  NZ,  Canada)  and  other 
external  organisations.  At  the  core,  watch  staff  contribute  to  the  maintenance  of,  effectively,  a 
COP  which  should  be  thought  of  here  as  a  combination  of  numerous  displays.  This  is  the 
articulation  of  the  accumulated  SA  delivered  to  the  rest  of  the  headquarters.  These  same  staff, 
fulfilling  either  a  J2  or  J3  function,  issue  routine  briefs,  drawing  from  the  COP.  The  J3  watch 
staff  are  also  responsible  for  initial  responses  to  events  outside  formal  standing  JTF  Areas  of 
Operation.  Elsewhere  in  the  J2,  analysts  provide  support  with  deeper  examination  of  events. 
We  may  identify  a  range,  then,  of  ‘organisation’  and  ‘product’  nodes  for  the  prospective  SA 
network,  which  we  list  in  Table  1. 


Table  1  Nodes  in  the  bipartite  network  after  aggregation  of  raw  data;  italics  indicate  organisational 
_ nodes  that  were  sampled. _ 


Organisations 

Characterisation 

Products 

Characterisation 

OCmnd 

Commander,  or 
function/section  Heads 

Pemail 

Formal  email  that  documents  information 
(rather  than  ad  hoc  communication) 

Pweb 

Documents  acquired  through  intranet 

Pbrief 

A  fused  brief  from  J3  and  J2  staff  (see 
below) 

PCOP 

Parts  of  a  COP  maintained  by  J3  and  J2 
staff 

OJ3W1 

Senior  Officers  in  J3 

Watch 

PJ3Brief 

Routine  Briefs  from  J3  Watch  staff 

OJ3W2 

Junior  Officers/NCOs  in 

J3  Watch 

PJ30rd 

Orders  issued  by  J3  Watch  staff 

OJ3WS 

Support  officers  in  J3 

Watch 

OJ3S 

Support  officers  in  J3 

PRpt 

Formal  Reports  by  military  staff 

OJ2W1 

Senior  Officers  in  J2 

Watch 

PJ2WBrief 

Routine  Briefs  by  J2  Watch  staff 

OJ2W2 

Junior  Officers/NCOs  in 

J2  Watch 

Padhoc 

Ad  hoc  document  or  PowerPoint  slide  to 
capture  ongoing  events 

OJ2A 

Analysts  in  J2  Staff 

PJ2AnBrief 

Analysis  Brief  by  J2  staff 

PThrAss 

Risk  Assessments 

OJ2S 

Support  officers/NCOs  in 

J2  staff 

Popen 

Open  source  information  -  internet 

OStrat 

Strategic  Level  Staff 
external  to  the  HQ 

Penv 

Reports  on  physical  environmental 
conditions 
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OExt 

Organisations  external  to 
Australia 

PExt 

Reports  from  organisations  external  to 
Australia 

OInt 

Other  intelligence 
organisations 

Pint 

Reports  from  other  intelligence 
organisations 

OJTF 

JTF  HQs 

PJTF 

Brief  or  formal  signal  from  JTF  HQ 

OSSl 

Single  Service  (Army, 

Navy,  Air  Force) 

Pdef 

Signals  from  Single  Service  units 

OWOG 

Non-defence  government 
departments 

PWOG 

Reports  from  non-defence  government 
departments 

5.  Data  Collection  Methodology 

In  adapting  propositional  networks  to  SAWN,  two  points  of  caution  were  addressed.  The  first 
is  that,  when  natural  language  is  used  to  represent  connections,  words  can  often  mean 
different  things  to  different  people  so  that  terms  in  questions  were  carefully  defined  and 
trialled  across  subjects.  The  second  issue  is  in  the  question  of  when  one  stops  building  the 
network,  namely  determining  when  it  is  complete.  To  that  end  a  clear  boundary  was 
established  to  only  include  the  differing  SA  perspectives  of  the  J2  and  J3  functions  of  the 
headquarters. 

We  thus  designed  a  survey  with  questions  designed  to  progress  incrementally  through 
Endsley’s  SA  levels  from  perception  through  comprehension  to  projection,  and  to  elicit  the 
distributed  nature  of  SA  measured  through  use/produce  and  interactions.  The  survey  then 
comprised  a  set  of  24  questions  related  to  products  (P)  and  organisations  (O).  The  ‘Products’ 
questions  ascertain  the  role  of  products  that  an  individual  prepares  for  others  and  those  that 
are  relied  on  for  that  individual’s  SA.  The  ‘Organisations’  questions  look  at  the  relative 
importance  of  formal  and  informal  interactions  with  groups  and/or  individuals  in 
organisations  external  to  the  subject  for  their  SA.  The  Survey  design  for  ‘steady-state’  and 
‘Crisis  Scenario’  snapshots  is  depicted  in  Figure  4,  which  indicates  the  sets  of  words  used  in 
the  questions  (“draw  attention”,  “identify”,  and  “understand  history”,  for  example). 


SAWN  Survey  Structure 

Sense^naking  during  a  72  hour  hatile  rtiyttmi 


Figure  4  SAWN  Survey  design 


For  the  ‘steady-state’  (or  ‘baseline’)  snapshot,  the  survey  was  administered  electronically  via 
Excel  attachment  to  an  email.  Participants  were  asked  to  consider  and  briefly  describe 
the  most  significant  events  of  the  past  36  hours  from  the  time  of  answering  the  survey 
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but  considering  their  anticipated  activity  for  the  next  36  hours.  They  were  then  asked 
to  choose  from  a  drop  down  list  up  to  four  products  and  organisations,  respectively,  in  order 
of  relevance  to  a  given  SA  activity. 

For  the  ‘Crisis  Scenario’  snapshot,  participants  were  individually  interviewed.  Following  an 
immersion  in  a  set  of  events  and  sequence  of  tasks  requiring  a  COP,  which  themselves  were 
derived  in  a  separate  scenario  workshop  (to  be  explained  below),  participants  were  asked  to 
select  products  they  would  use/produce  and  to  nominate  organisations  they  would  interact 
with  to  achieve  their  SA  or  support  someone  else’s  SA. 

Based  on  evidence  in  the  literature  that  SA  is  impacted  on  by  aspects  of  cognitive  workload 
[Endsley  2000,  Saner  et  al.  2009,  Gryszkiewicz  &  Chen  2012],  we  probed  this  issue  in  the 
survey.  We  adapted  the  National  Aeronautics  and  Space  Administration  Task  Load  Index 
(NASA-TLX),  a  validated  research  instrument  [Human  Performance  Research  Group  1988]. 
The  participants  were  asked  to  rate  their  perceived  levels  of  workload  in  terms  of  mental  and 
temporal  demands,  their  familiarity  with  the  task(s),  level  of  frustration,  fragmentation/effort 
required  to  perform  them,  and  to  rate  their  performance.  The  responses  were  recorded  on  a  7- 
point  scale  where  1  represented  the  lowest  and  7  the  highest  workload  level.  We  do  not 
present  results  of  these  aspects  in  this  paper. 

In  both  survey  and  interview,  minimal  demographic  data  was  collected,  namely  affiliation, 
rank,  length  of  service  and  posting.  The  latter  data  provided  indication  of 
expertise/knowledge,  important  elements  of  individual  SA.  To  the  ‘steady-state’  survey,  38 
responses  were  received,  with  17  from  J3,  and  21  from  J2.  The  ‘Crisis  Scenario’  interview 
involved  27  participants  with  15  and  12  from  J3  and  J2  respectively. 

6.  Developing  a  Scenario  based  Task  Model 

In  order  to  analyse  the  generation  of  SA  in  a  possible  future  crisis  scenario  we  needed  to 
understand  what  tasks,  information  and  intelligence  would  be  required.  To  this  end  we 
developed,  in  consultation  with  senior  J2  and  J3  staff,  a  scenario  description  covering  a 
hypothetical  event,  a  timeline,  and  a  range  of  constraints  and  assumptions.  The  scenario 
description  used  the  constraints  to  push  the  boundaries  of  the  study  participants  such  that  the 
tasks  and  information  needed  through  the  scenario  were  non-routine.  This  description  was 
distributed  to  military  SMEs  who  were  invited  to  a  knowledge  elicitation  activity  in  a  DSTO 
Operational  Test  &  Evaluation  facility  [TTCP  2006].  Through  this,  the  tasks,  information  and 
intelligence  requirements  through  the  period  of  the  scenario  were  articulated.  Participants 
were  drawn  from  planning,  operations,  intelligence,  and  single-service  units  in  the 
headquarters.  A  combination  of  two  dialogue  methods  was  used  to  elicit  these  requirements: 
Euture/Backwards  (EB)  and  the  aforementioned  CDM. 

EB  is  a  technique  that  facilitates  planning  in  complex  operational  environments  by  widening 
the  range  of  requirements  and  perspectives  of  a  diverse  group  of  players  [Cognitive  Edge 
2012].  The  EB  process  began  with  participants  determining  the  desired  end  state  to  be 
achieved  at  the  end  of  the  72  hours  of  the  scenario  development  and  then  delineating  critical 
decisions/actions  needed  (decisive  points)  to  take  place  in  order  to  achieve  that  end  state  and 
determining  when  they  should  occur.  The  EB  technique  resulted  in  obtaining  discrete  time 
frames  punctuated  by  the  decisive  points  which  were  used  to  determine  tasks  and  COP 
requirements  to  carry  them  out.  A  modified  CDM  [Klein  et  al.  1989]  was  employed  as  the 
dialogue  method  to  elicit  these  needs,  principally  because  of  its  role  in  DSA’s  Event  Analysis 
for  Systemic  Teamwork  (EAST)  methodology  [Walker  et  al.  2006].  At  its  core,  CDM  is  a 
retrospective  interview  that  applies  a  set  of  cognitive  probes  to  actual  non-routine  incidents 
requiring  expert  judgement  or  decision  making  [Klein  et  al.  1989].  It  asks  participants  to 
recollect  an  incident,  in  a  narrative  or  unstructured  manner  and  then  provides  a  set  of 
probes/questions,  to  understand  what  specific  decisions  were  made  through  the  course  of  the 


8 


incident.  Indeed,  as  noted  in  the  literature,  CDM  was  particularly  useful  because  of  the  varied 
expertise  of  the  participants  of  the  workshop.  Furthermore,  CDM  was  appropriate  because  the 
scenario  contained  several  non-routine  incidents,  for  which  each  subject  had  separate  real 
experience.  These  incidents  were  then  conflated  and  timed  to  occur  within  a  short  period 
of  time  to  require  rapid  decision-making.  CDM  was  modified  through  the  wording  of  probes 
to  account  for  the  hypothetical  nature  of  the  scenario  as  the  specific  sequence  of  events  had 
never  been  encountered  by  the  study  participants. 

As  mentioned  previously,  FB  was  used  to  identify  the  decisive  points  of  the  narrative.  From 
here,  the  workshop  facilitator  guided  each  of  the  participants  to  describe  the  events  and  tasks 
that  would  occur  between  each  decisive  point.  Finally,  specific  probe  questions  were  asked  to 
gain  the  information  and  intelligence  required  to  support  each  of  the  tasks/events.  The 
combination  of  FB  and  CDM  also  allowed  probing  of  participants  in  a  way  that  was 
deliberately  unfamiliar  to  them,  thus  requiring  greater  thought  in  developing  their  responses. 

During  the  workshop  we  used  the  SimVision  simulation  software  [Jin  and  Levitt  1996;  Levitt 
2004]  to  capture,  in  real-time,  the  participants’  responses.  ‘Swim-lanes’  for  each  of  the 
participant  units  were  placed  within  the  display  pane  of  the  software.  As  participants 
described  the  tasks  they  would  perform,  icons  representing  tasks  were  placed  in  the 
nominated  swim  lane.  Once  all  tasks  for  a  given  time  period  had  been  identified,  participants 
then  nominated  the  information  and  intelligence  required  for  the  COP  for  each  task.  These 
were  in  turn  represented  on  the  display  using  a  specific  icon.  Tasks,  Decisions  and  COP  items 
were  then  linked  in  a  notional  workflow.  The  SimVision  template  is  shown  in  Figure  5. 


]  ^  \\  ftaram  Tasks  etc  A  CDP  / 

Figure  5  SimVision  Task/Intelligence/Information  Template 


The  output  then  was  a  list  of  tasks  and  associated  COP  requirements  for  the  crisis  event 
arranged  around  four  Critical  Decision  Points  (CDPs)  with  approximately  80  associated  tasks, 
a  tenth  of  which  were  on-going  tasks  throughout  the  period  of  the  scenario. 

During  the  workshop  34  COP  requirements  were  identified  to  successfully  respond  to  the 
scenario.  The  sources/producers  of  this  information  were  then  separated  into  Internal  and 
External  Providers.  The  results  of  this  were  then  synthesised  into  a  short  document  and  a  quad 
chart  summarising  the  key  events,  tasks  and  COP  needs  through  each  of  the  four  decision 
phases.  This  document  was  distributed  to  participants  before  the  ‘crisis  scenario’  interview. 
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During  the  interview,  a  summary  of  the  scenario  via  the  quad  chart  was  presented  to  the 
participants  prior  to  being  asked  about  their  interactions,  use  and  generation  of  products. 

7.  Results 

Link  Weighting  and  Aggregation  scheme 

Each  link  between  an  individual  and  a  product  in  response  to  a  specific  question  was  assigned 
a  numerical  weight,  with  values  one  and  two  for  the  first  two  questions  at  the  lower  Endsley 
level  and  then  building  up  to  higher  values  as  explained  in  Table  2.  The  weighting  scheme 
was  chosen  such  that  nomination  of  a  given  individual  or  product  in  answer  to  multiple 
questions  at  lower  Endsley  levels  did  not  lead  to  the  same  or  greater  aggregate  weight  if  that 
individual  or  product  was  nominated  in  answer  to  a  single  question  at  a  higher  Endsley  level. 
Thus,  even  if  a  product  was  nominated  in  answer  to  all  questions  at  Endsley  Levels  I  and  II 
(aggregate  weight  of  66.0),  its  weight  would  still  be  less  than  if  it  was  nominated  just  once  in 
answer  to  a  question  at  Level  III  (minimum  of  68.0).  In  this  way  it  became  clear  where  the 
predominant  contribution  in  terms  of  Endsley  levels  lay  for  each  individual  or  product.  Next, 
the  results  for  multiple  individuals  were  aggregated  into  the  one  ‘organisation’  node,  because 
of  both  people  rotating  into  the  same  specific  role  in  a  shift  roster  (in  the  raw  data)  and  the 
aggregation  of  roles  into  the  generic  nodes  of  Table  1.  Here  we  average  by  summing  the 
weights  from  those  individuals  being  aggregated  who  have  nominated  a  specific  product  or 
organisation  and  dividing  by  the  number  in  that  aggregated  node  that  nominate  the 
product/organisation.  For  example,  if  there  are  four  people  in  a  shift  role  but  only  three  of 
them  nominate  a  certain  artefact  then  the  sum  of  the  weights  of  the  links  to  that  artefact  is 
divided  by  three. 

Colour  scheme 

As  explained,  data  was  captured  with  12  questions  refining  the  three  Endsley  levels  into  finer 
gradations.  The  resulting  networks  are  quite  complex  if  shown  to  that  level  of  fidelity. 
Therefore,  to  provide  a  more  readily  understandable  view  we  aggregate  the  data  according  to 
the  nodes  in  Table  1  and  combine  the  weights  to  the  three  Endsley  levels. 


Table  2  Colour  scheme  for  nodes  and  links  in  SAWN  diagrams 


Node 

Node  Colour 

Link  Endsley  Level 

Link  Colonr 

Sampled 

organisations 

Purple 

I:  Perception  (Qsl-3) 

Weights:  1.0,  2.0,  3.0 

Green 

Non-sampled 

organisations 

Orange 

II:  Comprehension  (Qs  4-8) 

Weights:  8.0,  10.0,  12.0,  14.0,  16.0 

Blue 

Products 

Green 

III:  Projection  (Qs  9-12) 

Weights:  75.0,  82.0,  68.0,  82.0 

Red 

We  also  aggregated  the  numbers  of  links  per  SA  level  according  to  the  scheme  in  Table  3. 

Table  3  Distribution  of  weights  according  to  bins 


Level  of  Bin 

Weight  Range 

1 

0-2.66 

2 

2.66-5.33 

3 

5.33-8 

4 

8-16 

5 

16-24 

6 

24-32 

7 

32-40 

8 

40-48 

9 

48-56 

10 

56-64 

11 

64-162 

12 

162-260 

13 

260-358 

10 


Networks 

We  present  the  resulting  SAWN  diagrams  (generated  using  NetMiner)  shrunk  in  size  but 
placed  side  by  side  to  provide  a  comparison  of  their  structure  between  ‘pull’  and  ‘push’  and 
between  steady-state  conditions,  Figure  6,  and  crisis  conditions,  Figure  7.  In  the  Appendix 
these  diagrams  are  expanded  in  size  where  they  may  be  examined  in  greater  detail.  To  allow 
the  interactions  between  the  sampled  staff  to  be  most  visible  we  arrange  those  nodes  on  the 
outside,  products  interior  to  them  and  external  nodes  (counter-intuitively)  at  the  centre  of  the 
network.  Colours  for  nodes  and  links  are  as  explained  in  Table  2. 


Figure  6  Situation  Awareness  Weighted  Networks  for  Steady-State  activity: 
pull  (left),  push  (right).  External  organisations  (orange  nodes)  are  arranged  in  the  centre, 
products  (green  nodes)  in  the  next  layer,  and  sampled  organisations  (purple  nodes)  in  J2  and  J3 
in  the  outer  ring.  Green  links  are  Perception,  Blue  links  Comprehension  and  Red  links  are 
Projection.  For  pull  links  the  point  of  the  arrow  is  on  the  node  that  draws  the  data  in  with  the 
link  colour  showing  the  SA  attributed  to  the  interaction  by  the  pulling  node;  for  push  links  the 
point  of  the  arrow  is  on  the  destination  with  the  colour  of  the  arrow  showing  the  SA  attributed  to 

the  interaction  by  the  source. 


Figure  7  Situation  Awareness  Weighted  Networks  for  Crisis  activity: 
pull  (left),  push  (right).  External  organisations  (orange  nodes)  are  arranged  in  the  centre, 
products  (green  nodes)  in  the  next  layer,  and  sampled  organisations  (purple  nodes)  in  J2  and  J3 
in  the  outer  ring.  Green  links  are  Perception,  Blue  links  Comprehension  and  Red  links  are 
Projection.  For  pull  links  the  point  of  the  arrow  is  on  the  node  that  draws  the  data  in  with  the 
link  colour  showing  the  SA  attributed  to  the  interaction  by  the  pulling  node;  for  push  links  the 
point  of  the  arrow  is  on  the  destination  with  the  colour  of  the  arrow  showing  the  SA  attributed  to 

the  interaction  by  the  source. 
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We  observe  a  number  of  features  in  these  diagrams.  First,  for  both  steady-state  and  crisis, 
there  are  more  links  for  pull  than  push:  our  subject  organisations  are  genuinely  consolidating 
information.  Second,  the  pull  data  is,  generally,  at  low  levels  of  SA  and  push  is  at  higher 
levels  (there  are  more  green  links  in  the  left  hand  networks  and  more  red  links  for  the  right 
hand  networks):  thus  the  subjects,  through  the  consolidation  of  information,  are  value-adding 
SA. 

These  features  can  be  seen  alternatively  by  counting  the  numbers  of  links  per  SA  ‘sub-level’ 
according  to  the  13  point  scheme  in  Table  3  and  plotting  a  histogram,  as  shown  in  Figure  8. 
We  see  there  that,  for  both  steady-state  and  the  crisis,  the  number  of  links  at  low  levels  of  SA 
(green  areas)  is  more  for  ‘pull’  than  ‘push’  for  levels  and  more  for  ‘push’  than  ‘pull’  for 
levels  12  and  13.  Also,  the  larger  number  of  SA  sources  for  the  crisis  scenario  is  evident, 
particularly  at  the  thresholds  for  Endsley  Levels  II  and  III;  for  example,  at  Level  II  more  than 
140  links  to  sources  were  identified  in  the  data. 

In  fact  there  is  a  significant  literature  on  this  type  of  behaviour.  In  recent  decades,  scholars  in 
interpersonal  communication  have  shown  interest  in  understanding  information  seeking 
behaviour  during  crisis  or  emergency  situations.  A  prominent  theory  here  is  ‘uncertainty 
reduction  theory’  which  posits  that  people  engage  in  intensified  information  seeking  activity 
to  alleviate  uncertainty  when  they  are  under  threat  or  facing  a  crisis,  perceived  or  actual 
[Berger  &  Calabrese  1975,  Afifi  &  Weiner  2002].  Overall,  the  research  indicates  that  in 
conditions  of  high  uncertainty  and  task  complexity  people  seek  more  information  from 
different  sources  for  decision  making  than  for  routine  tasks  and  they  show  preference  for 
verbal  as  opposed  to  written  media  [de  Alwis  et  al.  2006].  Our  data  is  consistent  with  this 
research  as  during  the  scenario,  the  participants  reported  far  greater  need  for  products  and 
interpersonal  interactions. 

We  also  observe  in  the  networks  that  higher  ranked  staff  (OJ3W1,  OJ2W1)  generate  higher 
levels  of  SA  than  lower  ranks,  and  in  turn  this  high  level  is  provided  to  commanding  officers 
(OCmnd).  This  is  consistent  with  the  organisational  design  of  these  units,  which  work  to  a 
standard  military  hierarchy.  However,  we  also  observe  team  behaviour  seen  in  the  self-loops 
in  both  Ligure  6  and  Ligure  7,  where  SA  is  passed  between  individuals  in  the  same  role 
through  a  shift  handover  or  within  teams. 

The  networks  also  expose  differing,  even  conflicting,  perspectives  on  the  SA  value  that 
individuals  or  products  provide.  Lor  example,  in  the  steady-state  pull  data  (Ligure  6,  left)  we 
observe  that  the  links  between  OJ2W1  and  OJ2W2,  and  OJ2W1  and  OJ3W1,  each  change 
colour.  The  colour  attached  to  the  arrowhead  indicates  the  level  of  SA  that  the  node  at  the 
point  of  the  arrow  (for  ‘pull’  data)  receives  from  the  adjacent  node.  Thus  OJ2W1  receives 
Perception  from  their  subordinate  OJ2W2  (the  green  arrowhead),  and  OJ2W2  receives 
Projection  (red)  from  their  superior  officers  OJ2W1,  a  reasonable  state  of  affairs. 
Contrastingly,  OJ3W1  receive  Perception  from  OJ2W1  while  the  latter  receives 
Comprehension  from  the  former.  However,  looking  at  the  out/push  data  for  steady-state 
(Ligure  6,  right)  we  see  clear  red  arrows  between  OJ2W1  to  OJ3W1:  both  believe  they  have 
pushed  Projection  levels  to  the  other.  Observe  in  the  crisis  scenario  (Ligure  7,  left)  that  this  is 
quite  different:  OJ3W1  expects  to  receive  Projection  from  OJ2W1,  while  the  latter  only 
expects  Comprehension  from  the  former.  Thus  the  SAWN  representations  enable  a  focus  on 
possibly  conflicting  perspectives  with  the  scope  for  enhancing  these  relationships  through 
exercises  and  experimentation. 
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Figure  8  Histograms  of  numbers  of  links  per  SA  level  for  steady  state  (top,  ^baseline’)  and  crisis 
conditions  (bottom,  ^scenario-based’).  In  (dark  grey  bars)  and  Out  (light  grey)  links  are 
distinguished.  Also  the  backgrounds  are  coloured  according  to  the  three-level  SA  model: 
green=perception,  blue=comprehension,  red=projection. 


Quantitative  Analysis  of  S A  flow.  The  application  of  weighted  bipartite  networks  now  permits 
the  use  of  novel  metrics  to  assess  the  flow  of  SA.  We  describe  two  of  these  here,  the  notions 
of  nestedness  and  SA  gradients,  and  give  a  numerical  example  of  the  latter. 

Nestedness  arises  in  environmental  modelling  to  analyse  the  inter-dependence  of  species 
communities,  for  example  plants  and  animals,  in  an  ecosystem  [Almeida-Neto  et  al.  2008], 
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and  is  the  degree  to  which  poorly  interacting  species  couple  with  highly  interacting  species. 
Representing  the  bipartite  network  as  a  matrix  of  ones  and  zeroes,  a  nested  species  network 
would  take  the  form  shown  in  Figure  9,  with  many  filled  cells  in  the  top  left  corner  and  many 
empty  cells  in  the  bottom  right. 
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Figure  9  Example  of  a  highly  nested  plant-animal  species  network:  a  black  cell  means  there  is  an 

interaction  between  an  animal  and  a  plant. 

Thus,  in  a  nested  network  there  will  be  found  a  core  of  high  degree  nodes  interacting  with 
each  other  but  also  with  a  set  of  low  degree  nodes  (the  latter  therefore  do  not  interact  with 
each  other).  Mathematically  this  is  formalised  in  the  ‘Nestedness  metric  based  on  Overlap  and 
Decreasing  Fill’  (NDOF).  This  metric  has  recently  been  generalised  to  weighted  networks 
[Almeida-Neto  &  Ulrich  2011],  but  still  relies  heavily  on  a  strong  contrast  between  degrees  in 
the  bipartite  graph  as  a  signal  of  nestedness. 

In  fact  our  SAWN  data,  if  we  focus  on  the  pure  bipartite  aspects  of  organisation  to  product 
interactions,  does  not  exhibit  strong  nestedness:  though  clearly  team  leaders  such  as  OJ3W1 
and  OJ2W1  are  more  connected  than  others  (reflecting  the  hierarchy)  there  are  no  subject 
‘organisational’  nodes  that  lack  connections  to  products.  This  potentially  reflects  a  degree  of 
self-synchronising  behaviour  within  the  set  of  J2  and  J3  staff  sampled;  everyone  plays  a  role 
in  seeking,  analysing  and  generating  information.  If  we  had  extended  the  sample  to  higher 
ranks  (those  in  the  OCmnd  node),  things  might  look  otherwise  where  many  senior  ranking 
officers  may  rely  entirely  on  a  verbal  brief  from  their  subordinates.  Thus,  we  hypothesise  that 
SA  weighted  networks  with  high  nestedness  will  reflect  strongly  hierarchical  behaviour. 


The  SA  gradient  measure  builds  on  paths  and  reachability  in  SNA  [Wassermann  and  Faust 
1994].  First  we  can  identify  nodes  pulling  the  lowest  levels  of  SA  and  those  pushing  the 
highest  levels  of  SA.  We  then  identify  paths  between  them,  either  solely  through  other 
organisational  nodes  or  via  product  nodes.  Insights  may  be  gained  by  considering  both 
shortest,  average  and  longest  path  lengths.  Then  a  gradient  for  the  path  may  be  defined: 


path " 


(v 


^(push)  _^^(puU)\ 
hg _ iig  ) 


Here  we  average  over  both  pull  and  push  weights  for  the  nodes  a  along  the  path. 


Arguably,  in  a  crisis  the  shortest  paths  are  critical.  So  in  our  SAWN  data,  this  quantity  is 
straightforward  at  the  level  of  fidelity  of  Figure  7.  The  OJ2W2  node  pulls  data  at  the  lowest 
level  of  SA  (Perception)  in  the  crisis  scenario,  with  a  flow  through  the  OJ2W1  node  and  then 
to  the  critical  OJ3W1  node  which  pushes  at  the  highest  level.  Projection.  This  results  in  this 
simple  case  in  a  gradient  of 

A5  =  1.75 


In  both  cases  here,  because  the  number  of  our  subjects  in  the  study  is  relatively  small 
(especially  after  aggregation)  and  their  relationships  are  already  close,  the  quantitative 
analysis  of  the  network  data  is  simple.  But  in  larger,  more  distributed  organisations  where 
agility  may  be  a  greater  challenge,  such  quantitative  measures  will  be  valuable. 
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8.  Conclusions  and  Future  Work 

We  have  unified  two  leading  models  of  Situation  Awareness  and  integrated  the  resulting 
network  view  of  distributed  adding  of  value  into  a  coherent  data  collection  methodology.  We 
used  aggregated  data  collected  in  a  study  in  the  Australian  Defence  Force  to  populate  the 
model.  The  resulting  representations  show  behaviour  consistent  with  the  hierarchical  C2 
arrangements  of  our  test  subjects  and  the  literature  on  how  managers  engage  in  increased 
information  seeking  in  situations  of  high  uncertainty,  and  reveals  where  team  work  is  active. 
In  this  specific  application,  the  method  has  identified  the  difference  between  how  a  COP  is 
maintained  under  steady-state  conditions  and  how  personnel  expect  it  should  be  maintained  in 
a  crisis.  In  particular,  the  nascent  C2  relationship  between  OJ3W1  and  OJ2W1  was 
recognised  to  be  critical:  each  belong  formally  to  different  Branches  but  mutually  depend  on 
each  other  for  sources  and  fusion  of  SA.  On  this  point,  the  study  recommended  routine 
exercises  across  these  organisations  to  enhance  the  existing  informal  relationship  and 
facilitate  an  emergent  process  for  the  dissemination  of  SA  across  the  two  entities. 

In  more  general  terms,  this  method  may  confirm  that  existing  arrangements  are  functioning, 
or  reflect  possible  missing  links  or  process  steps;  such  ‘gaps’  can  further  be  measured  in 
terms  of  graph  theoretic  metrics,  such  as  those  discussed  in  this  paper.  SAWN  diagrams  can 
also  expose  both  conflicting  perceptions  about  the  value  of  particular  organisational  roles  or 
information  artefacts,  and  flows  and  progressive  value-add  of  Situation  Awareness  through 
organisations.  Ultimately,  comparing  against  normative  models,  the  representations  can 
identify  whether  the  flows  and  build-up  of  SA  enable  agility. 

A  challenge  for  the  future  is  to  streamline  the  means  of  administering  the  underlying 
questionnaire  of  the  methodology.  In  this  application  steady  state  data  was  acquired  by  an 
electronically  administered  MS  Excel  forms  survey  and  crisis  response  data  by  interview 
(with  an  analyst  populating  the  survey  form).  In  the  first  case,  respondents  found  the  survey 
challenging  because  of  subtle  differences  between  questions  reflecting  incremental 
progression  through  Endsley’s  SA  model.  Also  navigating  long  lists  of  products  and 
organisations  was  found  to  be  frustrating.  This  was  despite  breaking  up  the  form  into  logically 
discrete  parts,  and  a  range  of  ‘smarts’  in  the  survey  form  designed  to  provide  explanatory 
information.  These  challenges  were  overcome  in  the  interview,  with  the  analyst  explaining 
any  subtleties  and  guiding  the  eye  of  the  interviewee  across  the  lists  (colour  coded  according 
to  the  nature  and  source  of  organisations  and  information  artefacts).  However,  for  more  senior 
officers,  whose  roles  require  accessing  multiple  organisations  and  sources,  this  led  to  long 
interviews  -  up  to  90  minutes  in  some  cases.  Though  this  is  less  than  the  two  hours  cited  for 
CDM,  it  was  still  a  challenge  for  staff  dealing  with  ongoing  real  operations. 

The  model  and  methodology  can  be  extended  in  a  number  of  ways.  To  the  extent  that  the 
Endsley  levels  may  be  seen  as  a  decomposition  of  Boyd’s  ‘Orient’  stage  of  the  OODA  loop, 
the  methodology  may  be  applied  across  the  entire  spectrum  of  activities  in  Endsley’s  model 
(including  Decision  and  Performance  of  Actions)  to  give  a  fully  distributed  network 
representation  of  her  model.  Such  a  representation  would  then  be  applicable  across  all  aspects 
of  a  C2  organisation. 

In  the  case  discussed  in  this  paper  we  applied  the  model  to  an  in  vivo  organisation  in  real 
operational  circumstances  for  steady  state,  and  a  hypothetical  scenario  for  the  heightened 
state.  This  means  that  in  no  case  did  the  analysts  involved  in  the  study  have  access  to  ‘ground 
truth’  by  which  the  Situation  Awareness  generated  by  the  subjects  could  be  ‘validated’.  This 
is  not  a  flaw  of  the  model  as  such  but  the  nature  of  the  application  (itself  a  consequence  of  the 
problem  we  were  invited  to  address).  The  same  limitation  applies  to  self-rating  techniques 
such  as  SART.  But  SAWN,  in  integrating  different  perspectives  on  the  same  information 
product,  allows  for  some  degree  of  testing  or  triangulating  the  subjective  judgements  of 
operators.  SAWN  is  also  applicable  to  simulated  environments  such  as  human-in-the-loop 
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experiments  or  command  post  exercises.  In  that  context,  where  analysts  do  have  access  to 
‘ground  truth’,  a  more  thorough  validation  of  the  awareness  reported  by  an  operator  can  be 
achieved.  Furthermore,  in  keeping  with  the  DSA  and  WEST  methodology,  SAWN  can  be 
applied  post-event  as  well  as  in  steady-state  and  hypothetical  scenario  situations.  SAWN  can 
be  applied  to  non-military  environments,  such  as  emergency  management  organisations, 
which  frequently  step  between  steady  state  and  crisis  events;  this  provides  further  contexts 
for  method  validation. 
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Figure  10  SAWN  Steady-state  pull 
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Figure  11  SAWN  Steady-state  push 
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Figure  12  SAWN  Scenario  pull 
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Figure  13  SAWN  Scenario  push 
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SAWN  for  Steady-State  activity:  Push 
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SAWN  for  Crisis  activity:  Pull 
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Numbers  of  links  per  SA  level  for  steady  state 
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Key  relationship:  J2-J3  Watch  leaders  -  high  SA  nodes 
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Conclusions 

■  SAWN  -  Situational  Awareness  Weighted  Network  which 

-  Unifies  two  leading  models  of  Situation  Awareness 

-  Contains  a  distributed,  adding  of  SA  value  Network  View 

■  Key  results: 

-  Confirmed  SA  flows  consistent  with  intended  C2  structure  and  mission 

-  Identified  and  quantified  the  as-is  relationship  between  two  key  nodes  in  different 
Branches  for  generation  of  high  SA;  recommended  joint  exercises  to  enhance  this 
relationship. 

■  Well  developed  and  tested  data  collection  method  that  can  be  used 
within  an  operational  context;  time-consuming  but  faster  than  classic 
CDM. 

■  SAWN  is  flexible  enough  to  be  applied  post-event ,  steady-state  and 
hypothetical  scenario  situations;  extension  to  simulated  experiments  or 
CPXs. 
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